Feature: Add diskquota extension to gpcontrib#1490
Merged
tuhaihe merged 327 commits intoapache:mainfrom Feb 27, 2026
Merged
Conversation
c916aae to
92de2e8
Compare
leborchuk
approved these changes
Dec 24, 2025
Contributor
leborchuk
left a comment
There was a problem hiding this comment.
Checked it on my development ubuntu installation, all simple tests works fine
Here the output from examples
postgres=# select * from diskquota.show_fast_schema_quota_view;
schema_name | schema_oid | quota_in_mb | nspsize_in_bytes
-------------+------------+-------------+------------------
s1 | 17157 | 1 | 98304
my-ship-it
reviewed
Jan 23, 2026
Contributor
my-ship-it
left a comment
There was a problem hiding this comment.
Thank you for your contributions that enable Cloudberry to support more components. Is it possible for us to retain the previous commit history?
Member
Author
NP. Let me bring them back for a complete commit history. |
8270730 to
de212a4
Compare
Member
Author
Hi @my-ship-it PTAL again. Thanks! |
Smyatkin-Maxim
approved these changes
Jan 30, 2026
812f1cf to
fec5729
Compare
We have to create database 'diskquota' manually before running regression tests. This change helps resolve it by creating the database in the init stage.
* Fix stack full issue when flush to table_size * Restore optimizer after flushing * Resolve deadlock and add expected test result
Historically, diskquota fetches tables' size by applying pg_table_size()
on a given oid list. It may invoke pg_table_size() several times in one
transaction. In order to tolerate exceptions, we wrap it with PG_TRY()
and consume any errors throwed by pg_table_size() using FlushErrorState().
It has the following 2 potential risks.
1. Deadlock: Considering we have a table 't1' (reloid=oid1) and it has a
partition 't1_1_prt_11' (reloid=oid2).
When diskquota_fetch_table_stat(1, '{oid2, oid1}'::oid[]) is called
in session 'A', this UDF will invoke pg_table_size() on 't1_1_prt_11'
and 't1' respectively. When pg_table_size('t1_1_prt_11') throws an
exception, the AccessShareLock acquired from session 'A' will be held
until the whole transaction ends. At the same time, when another
session 'B' is performing 'ALTER TABLE t1', it will acquire the
AccessExclusiveLock on 't1' and 't1_1_prt_11' respectively. As a result,
the session 'A' is holding the AccessShareLock on 't1_1_prt_11' and is
waiting for AccessShareLock on 't1'; the session 'B' is holding the
AccessExclusiveLock on 't1' and is waiting for the AccessExclusiveLock
on 't1_1_prt_11'.
2. Error message is missing. We should preserve the error message by calling
CopyErrorData() before FlushErrorState() and elog(WARNING, error_messgae).
Co-Authored-by: Xuebin Su <sxuebin@vmware.com>
Co-Authored-by: Hao Zhang <hzhang2@vmware.com>
Currently, tests for appendonly relations is missing. This patch adds one test for that. Co-Authored-by: Xuebin Su <sxuebin@vmware.com> Co-Authored-by: Hao Zhang <hzhang2@vmware.com>
* Support get_relation_size_by_relfilenode() add unit_test for calculating relation size by relfilenode * Add UDFs to allow checking size of a table Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xing GUO <higuoxing@gmail.com>
Co-authored-by: hzhang2 <hzhang2@vmware.com>
This patch introduces isolation2 test framework to diskquota. We are able to write concurrent test cases for diskquota in future.
* Fix incorrect relation size for AO tables * Rename function to make it more consistent * Remove try-catch when stat a file * Add concurrent write test case for AO table
1. add relation_cache_lock 2. add relation_cache and relid_cache in SHM 3. add relation_cache test case Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com> Co-authored-by: Xing Guo <higuoxing+github@gmail.com>
…_committed_relation_from_cache() (#96) Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com>
Co-authored-by: hzhang2 <hzhang2@vmware.com>
* Fix test case bug: Drop test table after testing diskquota.relation_size(), because the relation_entry of the table in this test case will affect the next test case. Co-authored-by: hzhang2 <hzhang2@vmware.com>
* Add support for dispatching blockmap to segment servers. This patch adds support for dispatching blockmap to segment servers. The basic idea is that, we iterate over the active relations' oid, check that whether the relation's owner/tablespace/namespace is in one of the blockmap entries dispatched from diskquota worker from QD. If the relation should be blocked, we then add its relfilenode together with the relfilenodes of toast relation, toast index relation, appendonly auxiliary relations, appendonly auxiliary relations' indexes to the global blockmap on segment servers. This patch is a prerequisite of the hard-limit feature. This patch also introduces diskquota.blockmap view. User is able to query the current blocking status by 'SELECT * FROM diskquota.blockmap'. Co-authored-by: Xuebin Su <sxuebin@vmware.com> Co-authored-by: Hao Zhang <hzhang2@vmware.com>
This PR fixes incorrect JOIN condition in test_blackmap.sql. The relfilenodes are not always different across segments. Hence, we should add an additional JOIN condition to the test case or it will produce unstable results.
Add calculate_table_size() to calculate any table's size, including uncommitted table. 1. Monitor active table in master. 2. Replace relation_open() with SearchSysCache(), pg_appendonly and pg_index to fetch Form_pg_class of table and index, to avoid DEADLOCK. 3. Use DiskQuotaRelationCacheEntry to calculate table size, instead of pg_table_size(), to avoid DEADLOCK. Co-authored-by: hzhang2 <hzhang2@vmware.com> Co-authored-by: Xuebin Su (苏学斌) <sxuebin@vmware.com> Co-authored-by: Xing Guo <higuoxing+github@gmail.com>
This patch adds support for adding uncommitted relations to blackmap on segment servers. Most of the codes share similar logic with adding committed relations to blackmap. Test will be added in the next following commits.
Change the CI pipeline to release diskquota for GP7. TODO: - enable regress test on CI. - enable activatestandby test for GP7 on CI. - fix regress test for GP7. The view in GP7 will be treated as a relation by diskquota. After `CREATE EXTENSION`, the test case should execute `SELECT diskquota.init_table_size_table()` to make the diskquota.state ready on GP7.
Fix the following issues:
diskquota cannot correctly print a warning when calling
`create extension` in a non-empty database. The diskquota.state shows
whether the current database is empty. Previously, we updated the
status to diskquota.state during `create extension` and queried the
status in UDF `diskquota.diskquota_start_worker`. When querying the
database status, the relations whose `relkind in ('v', 'c', 'f')` are
skipped, while all relations are filtered when updating the database
status. This patch merges the two SQL statements to `INSERT RETURNING`
to solve this problem.
Benefit: we won't need to upgrade the minor version of diskquota when
changing the statement.
Remove useless test `test_update_db_cache`. The behavior of the
bgworker scheduler is changed, so this test is no longer helpful.
Update the upgrade test script `2.2_set_quota`. Since the above issue,
there should be `select diskquota.init_table_size_table()` in
`2.2_set_quota`.
Add upgrade test for gp7.
Fix upgrade test for gp6.
Fix the standby test on GP7.
All tables in diskquota use primary keys. Unique index tables are created for primary keys. Unique index tables can not be created for tables with the option appendonly=true. This issue only occurs in gp6 since `set gp_default_storage_options='appendonly=true';` is disabled in gp7. Solution: add `appendonly=false` to each table with the primary key. Co-authored-by: Evgeniy Ratkov <e.ratkov@arenadata.io>
During the upgrade procedure, when diskquota.so is not in shared_preload_libraries, check whether the current version is the final version: - If yes, throw an error to the user. - Otherwise, return from _PG_init(). Co-authored-by: Sasasu <i@sasa.su>
This GUC shows the max number of database diskquota can monitor.
- Fix the compilation issue caused by GP7. `libpq.so` needs to be linked. See https://github.com/greenplum-db/gpdb/pull/16234. - Add GP7 jobs back to the pipeline.
`diskquota.max_quotas` shows the max number of quotas on the cluster.
In the current version of diskquota, quotas means:
- role
- schema
- role_tablespace
- schema_tablespace
The number of quota =
role + schema + (role * tablespace) + (schema * tablespace).
) Fix the bug caused by #220: After the user removes any extension, the bgworker in the current database will be stopped.
Co-authored-by: Chen Mulong <chenmulong@gmail.com> Co-authored-by: Xing Guo <admin@higuoxing.com>
Merge all types of quota info maps into one in each bgworker.
The package name is incorrect when releasing package in Rocky9/RHEL9 platforms (<package>-<version>-unknown_x86_64.tar.gz). This patch fix it to <package>-<version>-rhel9_x86_64.tar.gz. Cherry-pick https://github.com/pivotal/timestamp9/pull/41
- DDL message is only used on master, so it is unnecessary to allocate the memory on segments. - The launcher shmem should not be initialized on segments.
refresh_rejectmap looks for a tuple using SearchSysCacheCopy1 which retrieves a copy of the tuple allocating memory for it. However, refresh_rejectmap didn't free these tuple copies after use. If lots of oids were passed, diskquota could work incorrectly because of huge memory leak. This patch frees these tuples and prevents memory leaks.
fix: update actions Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Diskquota calculates sizes and stores information in the diskquota.table_size table periodically with a pause in diskquota.naptime, 2 seconds by default. If we restart the cluster during this pause, then diskquota will lose all changes that have occurred since the last save to the diskquota.table_size table. We could create temporary tables, wait when it will be flushed to diskquota.table_size table, restart the cluster, and diskquota would remember the information about the temporary tables. Or we could delete the tables, restart the cluster, and again diskquota will remember information about the deleted tables. This happens because at the start of the cluster, diskquota remembers all the information written to the diskquota.table_size table, but does not check that some tables may have already been deleted. As a solution, we invalidate diskquota.table_size during diskquota worker start in addition to pg_class validation. The remaining problem: the incorrect table size cannot be refreshed until the corresponding table becomes an active table. Solution: call diskquota.init_table_size_table().
Cleanup the invalid files after importing diskquota extension from upstream. The specific changes include: 1. Remove .github directory: The upstream GitHub Actions workflows are no longer applicable in the Cloudberry repository. 2. Remove .gitmessage, .editorconfig, .clang-format: Code style and commit templates should follow Cloudberry's main repository standards. 3. Remove SECURITY.md: Legacy security policy. 4. Remove concourse directory: Legacy CI scripts. This cleanup makes the extension structure cleaner and more consistent with other contrib modules.
Integrate diskquota extension into Apache Cloudberry build system and
adapt the codebase for Cloudberry 2.0+ (PostgreSQL 14 based).
Main changes:
Build system integration:
* Add new Makefile for building with Cloudberry source tree
```
make
make install
make installcheck
make clean
```
* Update gpcontrib/Makefile to include diskquota in build and installcheck
* Simplify CMakeLists.txt by removing GP6 version conditionals
* Add PG_SRC_DIR availability check for isolation2 tests
Code modernization (remove GP6 compatibility code):
* Remove GP_VERSION_NUM < 70000 conditionals throughout codebase
* Replace deprecated APIs:
heap_open -> table_open,
heap_beginscan_catalog -> table_beginscan_catalog,
heap_endscan -> table_endscan, etc.
* Replace init_ps_display() with set_ps_display() for process status
* Replace StrNCpy() with snprintf() for safer string handling
* Remove WaitForBackgroundWorkerShutdown() polyfill (now in core)
* Remove MemoryAccounting_Reset() calls (removed in GP7+)
* Update tuple descriptor attribute access from pointer to direct access
Documentation:
* Rewrite README.md for Apache Cloudberry with updated build instructions
Other improvements:
* Update extension comment to be more descriptive
* Ensure postgres.h is included first in all source files
CI:
* add ic-diskquota to `build-cloudberry.yml` workflow
* For `build-deb-cloudberry.yml`, the installation and configure prefix
are not consisent, which results in the test error. Will add
ic-diskquota test back once updating the deb workflow.
See: https://lists.apache.org/thread/1zd80r1hvpwwh5fjd5yqgcc7sr4f27qr
Add diskquota extension license information to comply with Apache release requirements. Changes: - Add diskquota entry to top-level LICENSE file under Greenplum section - Create licenses/LICENSE-diskquota.txt with PostgreSQL License text - Add gpcontrib/diskquota/** to pom.xml excludes for apache-rat checks The diskquota extension is licensed under the PostgreSQL License, originally developed by Pivotal Software and VMware.
fec5729 to
4a607d4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #ISSUE_Number
What does this PR do?
Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance:
User-facing changes:
Dependencies:
Checklist
Additional Context
CI Skip Instructions